Significant advancements in imaging sensors, microscopes, digital cameras, and digital imaging devices coupled with fast CPUs, high speed network connections and large storage devices enable broad new applications in the field of image recognition. The image recognition field includes a huge and broad range of practical activities including military and defense, biomedical engineering systems, material sciences, drug discovery, medical diagnostics, health monitoring, surgery, intelligent transportation systems, electronics manufacturing, robotics, entertainment and security systems. Image recognition applications entail the preprocessing and enhancement of images, definition of objects in images (image segmentation), calculation of whole image and object measurements, derived measurements and measurement statistics, data visualization, analysis, modeling and knowledge discovery from the measurements and statistics, and the classification of object subsets and/or the creation of image based decision analytics such as automated defect inspection systems, disease diagnosis systems, and pharmaceutical assay systems in early drug discovery. The robust encoding of processing rules and procedures into application recipes for high volume execution can be facilitated by machine learning technologies (Lee S J, Oh S, Huang C, 2003. Structure-guided automatic learning for image feature enhancement. U.S. Pat. No. 6,507,675; Jan. 14; Lee S J, Owsley L, Oh S, 2005. Online learning method in a decision system. U.S. Pat. No. 6,941,288; September 6; Lee S J, Oh S, 2004. Learnable object segmentation. United States Patent Application No. 20040202368; Oct. 14; Lee S J 2004. Dynamic learning and knowledge representation for data mining. United States Patent Application No. 20040267770; Dec. 30; Lee S J, Oh S, 2005. Method of directed pattern enhancement for flexible recognition. U.S. patent application Ser. No. 11/301,292; filed Dec. 7) at many points in the application workflow.
Image recognition applications involve the creation, manipulation, interaction, and viewing of different levels of data elements such as multi-dimensional images, image and application metadata, object masks, object regions of interest (ROIs), measurement data, summary statistics, object class and group designation, and processing recipes. The handling of these complex elements is compounded by the increasing dimensionality of imaging applications. For example, image recognition applications in advanced microscopy imaging are often termed “six dimensional” (x, y, t, z, channel and position) (Andrews P D, Harper I S, Swedlow J R. 2002. To 5D and beyond: quantitative fluorescence microscopy in the postgenomic era. Traffic. Jan;3(1):29-36.)
To deal with the complexity, expert users use multiple software for various tasks in their image recognition application (e.g. one software for image segmentation and measurement, and a different software for data analysis). Others purchase application specific turnkey solutions from original equipment manufacturers, or tailored solutions from system configuration providers or value added resellers, which are limited to the specific applications that the solutions are designed for. The lack of generic, easy-to-use software for managing the complexity of image recognition applications has prohibited the widespread adoption of image recognition software by end users.
A generic human-computer interface that can efficiently and transparently manage image recognition application complexity through a single interface would enhance the ease-of-use of image recognition software and reduce the cost of adoption and deployment by ordinary (non expert) users. This will allow the widespread use of image recognition technology in broad applications.
Prior art solutions range from simple image processing toolboxes to turnkey, application specific, high throughput image recognition solutions. No product provides a generic, all in one image recognition graphical interface that simultaneously presents the multi-level application elements, alongside application state-dependent processing choices.
At the most basic, image processing toolboxes such as Reindeer Graphic's Image Processing Toolkit and ImageJ, a public domain image processing software provided by Wayne Rasband of the National Institutes of Health, provide functions for image enhancement, image segmentation and mask measurements and can be considered image recognition software. With these and similar tools, images, masks and measurements are all treated individually. No relationships are maintained between these application elements. They all appear in their own, distinct, graphical windows.
Good examples of more sophisticated, easy-to-use and general purpose microscopy image recognition software include Intelligent Imaging Innovation's Slidebook and Compix's SimplePCI. Slidebook has a Slide interface that allows for an intuitive presentation of multi-dimensional images, and image derived object masks are maintained and presented alongside the multi-dimensional images in the graphical user interface (GUI). However, object measurements are treated as separate, unrelated elements, which are presented in separate windows and do not persist with the slide. They can only be saved to file. SimplePCI has an interface called the Data Document. It enables the presentation of some image recognition application elements along with processing tools. However, there are still several important drawbacks to this interface with make it difficult to use and non-ideal. Images are treated differently from the Data Document and have to be imported into the Data Document before the image recognition application can be completed. Basic image analysis tasks can be done on the image prior to import, while additional image recognition tasks can only be done in the Data Document. Second, the interface relies on a hierarchical folder structure which hides elements of the application from view. To navigate to various application elements or views of application elements users have to click through multiple levels of this hierarchy. It is desirable to present all application elements to the user simultaneously so that they can be accessed through a single click.
In addition, with the exception of SimplePCI, many of these general purpose image recognition software interfaces lack the capability to deal with large numbers of images and associated data. It is desirable to allow for the users to easily review image recognition elements for a large number of application elements, with all elements available in a single interface for easy and intuitive workflow.
The next type of prior art solution is tailored for high volume imaging applications. These include the Open Microscopy Environment's (OME) Shoola and Molecular Devices' MetaXpress. While these software enable high volume image and data handling and analysis, their interfaces are more task specific not generic for broad image recognition applications.
OME is a database driven, open source, open platform academic-industrial collaboration with the goal of creating standards for multi-dimensional microscopy image recognition database I/O and inter-application image and data exchange in support of high volume quantitative microscopy. In addition to developing the database backend and middle tiers, OME has also developed a web based image recognition GUI called “Shoola”. Shoola is a sophisticated GUI that covers the full breadth of image recognition applications in advanced microscopy research. However, Shoola provides too many element specific interfaces, making it difficult for novice users. Images are presented in the Image Viewer window, object data is presented in the ROI Analysis window, and object sets and subsets (classes) are presented in an image montage in the Hierarchy Browser. Furthermore, OME is paradoxically very general purpose but specific to imaging microscopy. This means that users have to configure their database, and set up database queries to drive the Shoola interface for their applications, but given the application logic inherent in both the database schema and the Shoola interface, the setup may not be useful for image recognition applications outside the specific field within quantitative microscopy. It is desirable to have an image recognition software and interface that is generic for all image recognition applications, require no database programming knowledge to populate, present all application elements in a single interface, even for high volume applications, in support of easy-to-use and intuitive workflow.
Molecular Devices' MetaXpress is an image recognition software for screening applications in early drug discovery. Similar to Shoola it provides support for high volume imaging and application element review. Unlike Shoola, it is a turnkey system providing application interfaces specific to drug screening applications (for example, images and data are arranged in well plate configuration) and requires no database configuration, though some database querying is required to populate the interfaces. The MetaXpress interface is non-ideal in many aspects. It requires some database query setup, and its GUIs are not relevant beyond drug screening applications. Application elements are not presented through a single graphical interface. The application elements cannot be freely modified, as access is driven by the application specific logic. The application specific logic drives the workflow through a series of wizards and GUIs that are specific to particular application elements. The user experience is non-intuitive and tiresome.